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[57] ABSTRACT 

A tuning control arrangement that dynamically con- 
trols the distribution of timing information in a distrib- 
uted digital communication system having standard 
nodes and prioritized master nodes. Reference timing 
signals to control the system are distributed from the 
highest priority master node currently operational 
Should a higher priority master node than die current 



master node become operational, the higher priority 
master node automatically takes control of the timing 
for die system. Each node is connected by links to at 
least one other node, and each node receives timing 
signals and a timing priority number (TPN) from all of 
the connected node. The TPN defines the number of 
links over which the timing signals travel to the receiv- 
ing node from the current master node. Each standard 
node selects timing signals and TPN from the link hav- 
ing the lowest TPN and transmits the selected TPN 
(after incrementing the selected TPN by one) and tim- 
ing signals on all connected outgoing links. When each 
master node becomes operational, the node transmits a 
TPN equal to M*N+ 1 where M is the priority number 
of each master node and N is equal to the number of 
links in the shortest path between the most distant nodes 
where distance is measured by counting the number of 
links between nodes. When a master node receives a 
TPN less than or equal to M*N, the master node enters 
the standby state and assumes the functions of a stan- 
dard node since the received TPN indicates that a 
higher priority master node is controlling the timing of 
the system. However, if a standby master node receives 
on all connected links TPNs equal to a predefined maxi- 
mum (which indicates that no master node is control- 
ling the system), the standby master node assumes the 
functions of an active master node and starts to control 
the timing of the system. 



23 Claims, 16 Drawing Sheets 
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out on links 101, 102 and 103. The result is that the TPN 

DISTRIBUTED TIMING RECOVERY FOR A numbers increase rapidly in magnitude until a predeter- 

DISTRIBUTED COMMUNICATION SYSTEM mined TPN m*™"™ is reached (which in the dis- 
closed system is 30). When the maximum is reached, 

TECHNICAL FIELD 5 backup master F detects this fact and begins to source 

This invention relates to a distributed digital system thahi & information for the system and to transmit a 

and more particularly, to improve facilities for recovery TPN of 1. The system will then arrive at a steady state 

and reconfiguration of timing between the nodes of the with the tuning information being supplied by backup 

digital system. in master F. 

10 When link 101 is repaired, master node A is placed 

BACKGROUND OF THE INVENTION back on line to supply timing information to the remain. 

It is necessary that all nodes of a distributed digital der of the system. Master node A also transmits a corn- 
system, such as a digital switching or transmission sys- mand to backup master node F to stop supplying the 
tern, have precisely synchronized timing so that voice timing information for the system illustrated in FIG. 1. 
and/or data signals can be transmitted error free be- 15 Upon receipt of this command, backup master node F 
twcen nodes. One way to achieve this timing precision ceases to supply the signaling information, 
is to transmit timing signals to all nodes from a master A problem with the aforementioned system is that it 
signal source so that the timing circuitry at all nodes is only allows for one master and one backup master node 
synchronized to the same signal. For reliability pur- which potentially limits the overall reliability of the 
poses, it is necessary that the system timing facilities 20 system if both the master and the backup master nodes 
have the capability of maintaining the internode syn- should fail. In addition, it is necessary for the master 
chronization at all times including circumstances in node to transmit a message to the backup master node 
which the master signal source may fail or become oncc the master node resumes supplying the timing 
otherwise unavailable. information for the system. Whereas this is not a severe 

U.S. Pat No. 4,736,393 discloses a timing control « ^^0^ it docs mcan that the master node does need 
arrangement the dynamically controls the distribution a reC0 rd of which node has been designated the backup 
of timing information in a distributed digital communi- mMt er node# jf f or some reason it is necessary to tempo- 
cation system. A reference timing signal is distributed ^ ^ backup master node from service, it is 
from a reference master node to all other nodes m the problem to designate another node as the backup 
system. Each node is connected by links to at least one 30 m ^ tef nQ ^ e 
other node and each node receives timing signals from 

all the links to which it is connected. Each node selects SUMMARY OF THE INVENTION 

one of these signals as its timing re^^scu^ ^ m ^ ^ ^ 

sf^T^^^ 35 = is acJved by ^t^J^T^ 

reference node. The path length is given by information ^ fo ? * J*** 

specifying the number of intermediate nodes through . ™P™ vcd tmimg facihtes allow for a plurality of pno- 

wttch the timing signal has traveled from the reference n ^ and provide that a lower priority 

node to reach the receiving node. The path length infer- automatically gives up controlling ; the timing for 

mation is conveyed in a timing priority number (TPN). 40 * e ^ m when a P nontv node ^omes opera- 

FIG. 1 illustrates the system of this patent which is tional- a . . 

capable of having one master and one designated Each of the nodes is interconnected to one or more 

backup master. In FIG. 1, the nodes are interconnected other nodes ™ activated, each operational 

by the indicated paths with the numbered arrows indi- master node transmits reference clock signals to control 

eating the TPN number transmitted over the path in 45 & e svstem and also a timing priority number (TPN). 

each direction. The number adjacent to each node indi- ™s TPN is equal to M*N+ 1 on links interconnecting 

cates the TPN generated by the node and transmitted ^ h master node to other nodes, where M is the prior- 

on all outgoing links. In FIG. 1, node A is the master to number of each master node and N is equal to the 

node; and node F is the backup master. The network number of links in the shortest path between the most 

condition portrayed in FIG. 1 indicates that the master 50 distant nodes where distance is measured by counting 

node is functioning properly to supply timing signals to *ne number of links between nodes. The highest priority 

the rest of the network. master node's M equals 0. Other than the master nodes, 

The system of FIG. 1 functions in the following man- each of the other nodes in the system selects clock sig- 
ner. Master node A generates and transmits a TPN of 1 nals and a TPN from an incoming link whose TPN has 
to node B, as indicated by the arrow pointing in the 55 the lowest magnitude. Each of the other nodes then 
direction of node B over path 101. Node B receives this transmits the selected clock signals and TPN (after 
TPN of 1, increments it by 1, and tranmits a TPN of 2 incrementing the TPN by one) to other nodes directly 
over paths 101, 102, and 103 to nodes A, D, and C, connected to each node. Each master node's timing is 
respectively. All of the nodes in the system perform controlled by clock signals received over a connecting 
similar incrementing and transmitting operations on 60 link if the TPN received on that link is less than M*N, 
received TPNs. By controlling the master nodes in this manner, the 

As long as master node A is active, the system re- highest priority operational master node is able to gain 

mains in the stable state illustrated by FIG. 1. However, control of the timing for the system either when the 

if link 101 fails, the remaining nodes in the system con- whole system is first made operational or when the 
tinue to increment and transmit the lowest received 65 higher priority master node becomes operational after 

TPN on the ongoing links. For example, when link 101 having been disabled. 

fails, the lowest TPN that node B receives is a 3. Conse- . If the current masster node controlling the system 

quently, node B increments this 3 by 1 and transmit a 4 should fail, the remaining nodes transmit increasingly 
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larger TPNs until the TPNs on all links have reached fare, node D increments the 2 by 1 and transmits the 

the predetermined maximum. When this state occurs, resulting TPN of 3 to each of nodes B, C, E and F. 

each non-master node selects an internal local oscillator Nodes F f G, H and I work in an analogous manner 

to provide timing for each individual node and trans- and receive and transmit the indicated TPNs. Each such 

mits a TPN equal to the predetermined maximum value. 5 node increments the lowest TPN it receives by 1 and 

In response to maximum-value TPNs on all links, each transmits this incremented TPN to each node to which 

operational master node transmits reference clock sig- it is directly connected. Thus, node I receives a TPN of 

nals to control the system and a TPN equal to M*N+ 1 5 &s well as a TPN of 6. Since 5 is the lowest value of the 

on connected links. However, each master node will two TPNs, node I increments 5 by 1 and transmits the 

switch to external clock signals from a connecting link 10 resulting TPN of 6 to each of nodes G and H. 

upon that link's TPN being less than M*N. This system The circuitry of each node operates in such a manner 

allows the highest priority operational master node to timing of the node is controlled from the signals 

gain control of the system's timing. on toe same path that transmits the lowest value TPN to 

the node. The details and operation of a node are illus- 

BREEF DESCRIPTION OF THE DRAWING 15 trated in FIGS. 3 through 8. Thus, node D receives 

This invention may be better understood by a reading of 2, 3, and 4. Since 2 is the lowest received TPN, 

of the following detailed description thereof with refer- node D uses the signals on path 202 to control its timing, 

ence to the following drawing in which: ***** ma nner, node I uses the signals on path 210 

FIG. 1 illustrates the network topology of a prior art to contro1 its timing rather than the signals on path 211. 

sytem* 20 Using this algorithm, each node receives its timing ref- 

FIG. 2 illustrates a network topology of one possible crencc cither directly or indirectly from first 

system of nodes in accordance with the invention; master node A over least number of P aths possible. 

FIGS. 3, 4, 5, and 6 when arranged and shown in ^ mast ff node A generates these timing signals from 

FIG. 7, disclose the details of the circuitry provided at „ £ mtemal 1 local °.^tor. Also, each node is prevented 

a nQc j e . 25 from supplying tuning back to itself through the net- 

FIG! 8 discloses the protocol used to transmit the ^ ^* c * ^ ould create loo P s ** d network 

TPNs over the system paths; Xm ^ s T ^£ 1 • *u ■ 

FIGS. 9 through 23 illustrate different states that the * to of .?W™ 

-t— — - , _ of rICj. 2 m accordance with the invention, the follow- 

system of FIG. 2 can assume dunns automatic reconfis- _ . , " . " , ' . 

7 *T * ~r j !r*r ^ , B 30 ing example is given to illustrate how master and non- 

uration of the system due to failure and subsequent ® *\ 6 .7 j . 7 \- 7 

reoair of master odes: d 4 master nodes function during automatic reconfiguration 

nr * TT /Je^i * * ■ n x. ^ c *u due to master nodes becoming inoperable, and subse- 

FIGS. 24 and 25 illustrate, tn flowchart, form the d blc> r ^ e ^ P flllIlta J M m p^s. 

manner m which the circuitry at each node operates to 9 thxon ^ lz ^ tetion8 of th £ nodes when ^ first 

!^/^!IT^ toto T!!!?T C, ^ 35 node Ms with all other nodes operational. Sec- 
tions and states that may be encountered by the node. ond> ^ example Ulustrates in p^g. ^ ^ u the 

DETAILED DESCRIPTION functions of the nodes when the second master node 
^ , . fails with the first master node inoperable but all other 
pe network topology of a system in accordance nodes operational . ^ examplc mustrates m 
with the present invention is djustratedin FIG. 2 where 40 FIGS. 15 and 16 the functions of the nodes when the 
the network comprises the nodes A, B, C, D E, F, G, H ^ mastcr ^ fails ^th the first and second master 
and I. The nodes are interconnected by the indicated nodes being inoperable but all other nodes operational, 
links/paths. Associated with each path are two TPNs Fourth, the example ulustrates in FIGS. 17 and 18 the 
transmitted over the path in each direction as indicated functions of the nodes when the second master node 
by the arrows. The number adjacent to each node indi- 45 becomes operational with the first and third master 
cates the TPN generated by the node and transmitted nodes inoperable but all other nodes operational. Fifth, 
on all of its outgoing links. me example ulustrates in FIGS. 19 through 21 the func- 
In FIG. 2, node A is the first master node, node H is n0 ns of the nodes when the first master node becomes 
the second master node, and node E is the third master operational with the 1 second master node operational, 
node. First master node A has the highest priority and 50 the third master node inoperable, and all other nodes 
an M equal to 0; second master node H has the second operational. Finally, the example illustrates in FIGS. 22 
highest priority and an M equal to 1; third master node and 23 the functions of the nodes when the third master 
E has the third highest priority and M equal to 2. The node beomes operational with all other nodes opera- 
network conditions as illustrated in FIG. 2 indicated tionaL 

that first master node A is functioning properly supply- 55 The system timing remains in the state portrayed by 

ing timing signals to the rest of the network. In this FIG. 2 as long as all elements of the network, including 

stable state, first master node A generates and transmits master node A and path 201 remain operable and supply 

a TPN of 1 to node B as indicated by the arrow pointing the required timing signals to node B. If path 201 fails, 

in the direction of node B over path 201. After node B node B no longer receives a TPN of 1 from node A. In 

receives this TPN of 1, node B increments the TPN by 60 this case, a TPN of 3 is the lowest value TPN received 

1 and applies the resulting TPN of 2 over paths 201, 202, by node B. Node B increments the 3 by 1 and generates 

and 203 to nodes, A, D, and C respectively. After node a TPN of 4 which is transmitted to nodes C and D. 

C receives the TPN of 2, node C increments the re- Each of the nodes C and D increment the received TPN 

ceived TPN by 1 and transmits the resulting TPN of 3 of 4 and transmit a new TPN of 5 back to node B. Node 

to node D over path 204. Node D receives a TPN of 2 65 B in turn increments 5 by 1 and returns a new TPN of 

from node B and a TPN of 3 from node C. Node D also 6 to nodes C and D. This process continues with respect 

receives a TPN of 4from each of nodes E and F. the to nodes B, C, and D so that increasingly higher value 

TPN of 2 is the lowest TPN received by node D; there- TPNs are generated. The remaining nodes of the net- 
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work follow in an analogous manner and they also tuning signals and transmits a TPN equal to 1. As illus* 

generate increasingly higher value TPNs. Finally, the trated in FIG. 19, both first master node A and second 

system reaches the state illustrated in FIG. 9 where all master node H axe now operational, each transmitting a 

nodes are transmitting and receiving a maximum TPN, TPN and timing signals to control the system. Third 

which illustratively may be 30. 5 master node E is not operational at this point 

Once any node receives the maximum TPN, the node FIG. 20 illustrates the next progression of first master 
does not increment the received TPN but continues to node A assuming control of the system since it is trans- 
transmit the tnnKimnm TPN on all outgoing links. In mitting a lower TPN than second master node H. The 
addition, the node relies on an internal local oscillator nodes in the system continue to accept the timing infor- 
fbr timing. Once the steady state condition illustrated in 10 mation from first master node A as each in turn receives 
FIG. 9 is obtained, both second master node H and third a lower TPN accompanying the timing information 
master node E transmit the necessary timing signals on from first master node A than the TPN accompanying 
their outgoing links to control the system. Second mas- the timing information from second master node H. 
ter node H transmits a TPN of 6 on outgoing links 209 Both the first and second master nodes continue to 
and 211. Third master node E transmits a TPN of 1 1. IS transmit internally generated timing signals and TPNs 

In general, the master nodes generate and transmit a until the state of the system illustrated in FIG. 21 oc- 

TPN equal to M*N+ 1 where M is the priority number curs. Once second master node H receives a TPN of 5 

of the master node. In the present example, first master from node G via link 209, second master node H stops 

node A, second master node H, and third master node E outputting internally generated timing signals and ac- 

have priority numbers equal to 0, 1, and 2, respectively. 20 cepts timing signals received via link 209 from first 

N is equal to the number of links in the shortest path master node A. At this point, all the nodes in the system 

between the most distant nodes where distance between are receiving the timing signals from first master node 

nodes is measured by counting the number of links A. 

between the nodes. N equals 5 for the present example. FIG. 22 illustrates the state of the system when third 

FIG. 10 illustrates the TPNs transmitted during the next 25 master node E becomes operational. Third master node 

TPN phase. FIG. 11 illustrates the second TPN trans- E initially transmits internally generated timing signals 

mission. Once the third master node E receives the and a TPN equal to 11 when it first becomes opera- 

TPN of 7 from node G, node E immediately stops trans- tionaL However, on the next TPN phase, third master 

mitring tuning signals, utilizes the timing information node E recognizes the TPN of 3 on link 205 and ceases 

received via link 208, and starts transmitting a TPN of 30 to internally generate timing signals and a TPN; and 

7+1 on its outgoing links as illustrated in FIG. 12. In third master node E begins to accept its timing signals 

general, master nodes assume the functions of a nonmas- from link 205 (supplied indirectly by first master node 

ter node when a received TPN is equal to or less than A). The system achieves a steady state as illustrated in 

M*N which indicates that a master node of higher pri- FIG. 23 where master node A supplies all the timing 

ority is controlling the system. Eventually, the system 35 signals. 

stabilizes to the state illustrated in FIG. 13 where sec- FIGS. 3, 4, 5 and 6, when arranged as shown in FIG. 

ond master node H supplies all of the timing informal 7, disclose the circuitry comprising a typical node of 

tion to all nodes of the system with the exception of FIG. 2. The node 300 on FIGS. 3 through 6 comprise a 

master node A which has been severed from the system plurality of line interfaces 301, 302, 401 and 402 each of 

due to a failure of link 201. 40 which terminates one of the links or paths to which the 

Once the system has achieved the steady state condi- node is connected.. The function of the equipment of a 

tion illustrated in FIG. 13, and second master node H node is to receive information from a connected link 

fails, the nodes increment the TPNs until each TPN and to apply information to the connected link. The 

equals to 30. At this point third master node E transmits bidirectional links on the left side of FIGS. 3 and 4 are 

timing information to control the system and a TPN of 45 designated LI, L2, L3, and L4. These links comprise the 

11. The system eventually stabilizes as illustrated in paths of FIG. 2 connected to the node portrayed by 

FIG. 14. FIGS. 3 through 6. For example, node D of FIG. 2 is 

If third master node E fails as illustrated in FIG. 15, connected to the four paths 202, 204, 205 and 206 and 
all operational nodes once again increment received therefore, these four paths correspond to the bidirec- 
TPNs and output these incremented TPNs on their 50 tional links designated LI through L4 on the left side of 
outgoing links. Since there are no longer any master FIGS. 3 and 4. Only nodes D and G of FIG. 2 are 
nodes capable of assuming the generation of timing connected to four paths. The remaining nodes are con- 
signals for the system illustrated in FIG. 15, the system nected to either one path, two paths or three paths. For 
eventually stabilizes as illustrated in FIG. 16. Each node a node connected to only one path, only one of the links 
then relies on its own internal local oscillator to gener- 55 LI through L4 would be connected to only one path, 
ate its timing signals and transmits a TPN equal to 30 on only one of the links LI through L4 would be con- 
all outgoing links. nected on FIGS. 3 and 4. Similarly, less than all of the 

Now assume that second master node H becomes links LI through L4 on FIGS. 3 and 4 would be used 

operational as illustrated in FIG. 17. When second mas- for the other nodes of FIG. 2 that are conntected to less 

ter node H becomes operational, it outputs a TPN of 6. 60 than four paths. 

The non-master nodes utilizes the timing signals from In addition to the link interfaces 301, 302, 401 and 
second master node H and eventually reach the stable 402, each node comprises a digital switching or trans- 
state illustrated in FIG. 18. In FIG. 18, the system uti- mission system 501, a clock system 503, and a node 
lizes the timing signals from second master node H to controller 601. The digital switching system or trans- 
achieve synchronization. Both the first and third master 65 mission system 501 comprises the equipments at the 
nodes are inoperable, .p FIG. 19 illustrates the state of node that receives the data 6t information, other than 
the system when first master node A first becomes oper- the TPN information, that is transmitted over the net- 
ationaL First master node A immediately sends out work between nodes. If the network of FIG. 2 repre- 
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sents a telephone network, then the element 501 would lot signals on incoming path R of link LI and applies 

comprises a digital switching system such as a digital them to the digital switching or transmission system 

central office or a digital PfiX wherein each central 501. Thus, system 501 receives all of the timesolt signals 

office or PBX serves a plurality of connected subscriber applied to receiver 503 while incoming TPN register 

stations. On the other hand, if the network of FIG. 2 5 310 receives only the timeslot 0 signals representing the 

represents a digital transmission system, then element TPN that is transmitted over path LI to the node rep re- 

501 would comprise a digital system serving a plurality sented by the circuitry of FIGS. 3 through 6. 

of digital transmission facilities. In addition, the net- Digital Switching or Transmission System 501 ap- 

work of FIG. 2 could represent a combination of PBXs plies the data it wishes to transmit to another node to 

and digital transmission facilities. 10 cable 517 and from there to the link interface and link 

Clock system 503 comprises a precision oscillator 505 that is to extend this data to the other node. Let it be 

and a timing source selector switch 504 which permits assumed that the data is to be transmitted over link LI. 

the timing of equipments at the node to be referenced to In this case, the data from system 501 is applied to path 

either the local precision oscillator 505 or to a clock 327 where it is inserted into a signal comprising a series 

signal recovered from the data applied to the node over IS of timeslots as on FIG. 8. Timeslot inserter 308 also 

any of the connected bidirectional links LI through L4. receives from element 311 the TPN that is to be trans- 

The node controller 601 is a microprocessor controlled mitted in timeslot 0. Inserter 308 combines the timeslot 
element having a microprocessor 610 and a plurality of data on path 327 with the TPN on path 329 and applies 

registers and timers which operate under microproces- the resultant signal stream to transmitter 306 for trans- 

sor control to control the clock system 503 so that the 20 mission over path T of link LI to the node connected to 

equipments of the node 300 operate under control of a other end of the link. The timeslot inserter receives the 

preferred clock signal in accordance with the present node clock signal on path 330. The clock signal on path 

invention. The operation of the node 300 and the node 330 is also extended to transmitter 306 to control its 

controller 601 is subsequently described in detail in operation and this clock signal is embedded in its data 

connection with the flowcharts of FIGS. 22 and 23. 25 stream for recovery at the node which terminates it. 

With respect to link interface 301, conductor R of The TPN that is to be transmitted from the node over 

link LI carries the signals applied to node 300 and these link LI is stored in register 620 and extended over path 

signals are applied to receiver 303. These signals are 332 to the outgoing TPN register 311 for line interface 

divided into discrete timeslots as shown in FIG. 8 Re- 301. The outgoing TPN information is inserted into 

ceiver 303 receives these timeslot signals and applies 30 register 311 and applied by this register over path 329 to 

them over its output on path 324 to timeslot extractor timeslot inserter 308. Inserter 308 inserts the outgoing 

307. Timeslot extractor 307 extracts the TPN informa- TPN into timeslot 0 and inserts the information on path 

tion from the bits of timeslot 0 and applies this informa- 327 into the remainder of the timeslots of each frame 

tion over path 328 to incoming TPN register #1 which generated by link interface 301. The timeslots of each 

is also designated as element 310. This TPN is applied 35 generated frame are applied from inserter 308 to trans- 
over path 325 from register 310 to the A register 616 of mitter 306 which in turn extends them over path T of 

node controller 601. Each of registers 616-619 com- link LI to the node, of the network connected to the 

prises A and B sections. The signal applied to receiver other end of link LI. 

303 is also extended over path 312 to timing recovery FIGS. 24 and 25 illustrate the program executed by 

element 305 which recovers the clock signal embedded 40 microprocessor 610 of FIG. 6 in controlling the various 

in the received data on the R path of link LI. This clock nodes to perform the task previously described. Each 

signal is extracted and applied over path 326 to contact node illustrated in FIG. 2 is capable of executing all 

2 of timing source selector switch 504 of clock system portions of the flowcharts shown in FIGS. 24 and 25. 

503. Recovered clock is also applied over path 323 to The master of non-master status is determined by the 

receiver 303. Clock maintenance information is applied 45 contents of registers 622, 623, and 655 of FIG. 6. The 

over path 331 to the receiver maintenance and interface utilization of other registers illustrated in FIG. 6 is 

registers 615. Path 331 comprises a maintenance bus and given in greater detail in U.S. Pat. No. 4,736,393 which 

one of its functions is to supply register 615 with infor- is hereby incorporated by reference. Each node first 

mation indicating whether receiver 303 is receiving reads the incoming TPNs utilizing block 2401 and then 

valid data and whether timing recovery element 305 is 50 checks to see whether the node is designated as a master 

receiving valid data and whether timing recovery ele- node by execution of decision block 2402. In the present 

ment 305 is successful in recovering good timing from example if the node is first master node A, then decision 

the signal received by receiver 303 from link LI. block 2403 determines whether the node is capable of 

Path 330 extends from the clock signal generator of driving the system. If the node is capable of driving the 

the clock system on FIG. 5 to the link interface units on 55 system, then blocks 2406 and 2409 supply the necessary 

FIGS. 3 and 4. On FIG. 3, path 330 connects to timeslot timing information to the network and outpulse a TPN 

extractor 307 and receiver 303. Path 330 carries the of 1. If first master node A is not capable of driving the 

clock signal that controls the operation of the node and network, then decision block 2404 determines whether 

therefore, the clock signal on path 330 controls the or not one of the other master nodes lis supplying timing 

receiver 303 and the timeslot extractor 307. The signal 60 information by examining the incoming TPNs to deter- 

from the. R lead of link LI is clocked into the receiver mine if any are less than 30. If another master node is 

303 using the recovered clock applied on path 323 and driving the system, then blocks 2408 and 2411 are exe- 

the incoming data on path 324 is clocked out of the cuted whereby first master node A derives its timing 

receiver 303 using the clock signals applied on path 330 from information on an incoming link and outputs a 

from the clock system 503 of FIG. 5 65 TPN equal to the incoming TPN plus 1. If no other 

The output of receiver 303 is also extended over path master nodes are applying timing signals to the net- 

324 and cable 516 to the input of the digital switching or work, then blocks 2407 and 2410 are executed whereby 

transmission system 501. This path carries all the times- first master node A relies solely on its own internal- 
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oscillator and transmits a TPN of 30 on all outgoing The system of FIG. 2 also functions by the non-mas- 

links. ter nodes and masters nodes functioning as non-master 

If the node implementing the flowcharts illustrated in nodes decrementing rather than incrementing received 

FIGS. 24 and 25 is not a master node, then blocks 2405 TPNs. The controlling master node generates a TPN 

and 2412 are executed. Decision block 2412 determines 5 equal to M*N+a— 1 where M and N are as previously 

if there is a node supplying tuning information to the defined and "a" is a predefined number. Advanta- 

system by checking to see if any of the incoming TPNs geously, **a" equals 20 for the system of FIG. 2. The 

are greater than or equal to 30. If an incoming link has highest priority master node's M equals the highest 

a TPN of less than 30, then blocks 2413 and 2414 are value of M. Advantageously, first master node's M 

executed whereby the node derives its timing informs- 10 equals 2 for the system of FIG. 2. For the system of 

tion from an incoming link and outputs a TPN equal to FIG. 2 if the controlling master node faded, the standby 

the lowest received TPN plus 1. master nodes start to control the system's timing when 

If the node is receiving a TPN greater than or equal received TPNs are less than or equal to a predefined 

to 30 on all links, which indicates that no master node is number which could advantageously be 0. A lower 

supplymgtinimgagiialstomenew 15 pr i or ity master node relinquishes tuning control to a 

must determine whether it is a second or third master priority master nodc when the lower priority 

node or a non-master node. Decision blocks 2415 and maste r node receives a TPN greater than M*N+a-2. 

2416 determine whether the node is a second or third while a specific embodiment of the invention has 
master node. If the node is a non-master node, blocks ^ disclosed, variations within the scope of the ap- 

2417 and 2418 are executed whereby the node simply 20 ded claimes m ^ d ^ contemplated, 
derives tuning information from its own oscillator and fa no mt ention of limitation to what is contained 
outputs a TPN equal to 30 on all its outgoing links. m ±e ab stract or ^ exact disclosure herein presented. 

Decision block 2415 toennmes whether t^node is a The abovCKjcscribe d arrangements are only illustrative 
f^J^^ m ^^J an ^J^f^ l P' „ oftheapphcationoftheprmciplesofthemvention.Itis 

t > maStCT n ^?L^ dCC1S1 ° n 25 to be expected that other arrangements will be devised 

block 2501 of FIG. 25 determines whether this master . . . ^\ . *u * j _^ r *t. 

node is capable of supplying the system. If the second * ^tailed in the art without departing from the 

master node is capable of supplying tuning to the sys- ^ a ^ ^ of the mventl ° n - 

tern, blocks 2502 and 2503 are executed. After execution * c * r . . . ^ . . ^ 

of these two latter blocks, the second master node re- 30 . t?****™ fo < w ma niultmode system of the type 

mains in a loop comprising blocks 2501, 2502, 2503, and m each node receives, from each other node to 

2504. This loop continues until decision block 2504 ** ch each * ***** connected, (a) a dock sign* and 

detects that an incoming link has a TPN less than N+ 1. mput timing priority number (TPN), 

This condition indicates that the first master node has m , ^ ch ^ node transn11 ^ to said each other 

once again started supplying the timing signals to the 35 < a > P^ c ^J^ v ^ clock * e 

system. When this condition is met, blocks 2508 and ^west associated TPN and (b) an output TPN, the 

2509 are executed TPNs being such that, at any particular tune, the clock 

If the second master node cannot supply timing to the from a particular node, a current one of a" plural- 
system, then decision block 2505 is executed to deter- *Y of "■fw nodes becomes distributed to each of the 
mine whether an incoming link has a TPN less than 30 40 othcr no<les 83 a current master clock signal, each of 
which, of course, indicates that another node is supply- said plurality of master nodes being prioritized in accor- 
in g timin g gignaltt tn thft nfttmnrV Tn th* prpgpnt system, dance with an assigned master priority number, the 
this other node would be the first or third master node, higher priority node having the lower master priority 
If this condition (all incoming TPNs S 30) does not number with the highest priority node having the low- . 
exist, then blocks 2506 and 2507 are executed. How- 45 master priority number, said apparatus including, at 
ever, if another node is supplying timing signals to the each individual one of said plurality of master nodes of 
network, then blocks 2508 and 2509 are executed. the system: 

If the node is not a second master node, control is means for operating said individual node in a master 
passed from decision block 2415 to decision block 2416 mode in which its output TPN is a predetermined 
which determines whether or not the node is a third 50 number which is a function of a piedetennined 
master node. If the latter condition is met, then decision mastr priority number assigned to that node, 
block 2510 of FIG. 25 is executed Decision block 2510 means for operating said individual node in a mode 
determines whether or not the node is capable of sup- other than said master mode in which its output 
plying timing signals to the system. If the node is capa- TPN is a function of the TPN associated with said 
ble of supplying timing signals to the system, blocks 55 particular received clock signal, 
2511 and 2512 are executed such that the node derives means for switching said individual node to its master 
timing information for the system from an internal local mode upon the disappearance of said current mas- 
oscillator and outputs a TPN of 1 1. The third master ter clock signal from said system, and 
node remains in the loop of blocks 2510, 2511, 2512, and means responsive to the receipt of a TPN which is 
2518 until either the first or second master node starts to 60 smaller than said TPN predetermined number for 
supply timing information to the system. Block 2518 thereafter switching said individual node back to 
determines when this condition occurs. If the third its other than said master mode, 
master node is not capable of supplying riming signals to whereby said multinode system converges to a stable 
the network, decision block 2513 is executed to deter- state in which an operative one of said master 
mine whether or not there is an incoming link that has 65 nodes having the highest priority operates in its 
a TPN less than 30. If this condition (at least one master mode as said current master node and each 
TPN <30) is met, then blocks 2516 and 2517 are exe- other of said master nodes operates in its other than 
cuted. Otherwise, blocks 2514 and 2515 are executed. said master mode. 
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2. The apparatus of claim 1 wherein said plurality of 
master nodes are prioritized; and 

said means for operating said individual node in said 
master mode comprises means for generating said 
predetermined number equal to M*N+i where i is 5 
a predefined number and M is the master priority 
number of said individual node and N is equal to 
the number of links in the shortest path between the 
most distant nodes where distance is measured by 
counting the number of links between nodes. 10 

3. The apparatus of claim 2 wherein said means for 
operating said individual node in said other than said 
master mode comprises means for adding i to said TPN 
associated with said particular received clock signal. 

4. An apparatus for controlling the timing of each 13 
node in a multinode system wherein a clock source in 
one of a plurality of prioritized master nodes provides a 
reference clock signal for all other nodes in said system 
and the higher priority node of said plurality of priorit- 
ized master nodes has the lower priority number with 20 
the highest priority node having the lowest priority 
number, comprising: 

means for interconnecting each of said nodes to one 
or more other ones of said nodes via links; 

means for transmitting clock signals between each of 25 
said nodes directly interconnected by any one of 
said links; 

means for activating one of said plurality of priorit- 
ized master nodes to provide reference clock sig- 
nals for controlling the timing of said multinode 30 
system; 

means for including a timing priority number (TPN) 
of M*N +i in the clock signals sent from said one of 
said plurality of prioritized master nodes to all of 
said nodes directly connected to said one of said 35 
plurality of prioritized master nodes by any one of 
said links where i is predefined number and M is the 
priority number of said one of said plurality of 
prioritized master nodes and N is equal to the num- 
ber of links in the shortest path between the most 40 
distant nodes where distance is measured by count- 
ing the number of links between nodes; 

means for operating all others of said nodes, other 
than said one of said plurality of prioritized master 
nodes, that receives a TPN so that all others of said 45 
nodes alter the received TPN by adding the num- 
ber i to the received TPN and transmitting the 
altered TPN with clock signals to each node to 
which it is directly connected by any one of said 
links; 50 

means for operating each node that receives more 
than a single TPN, other than said one of said plu- 
rality of prioritized master nodes, to select one of 
said received TPNs as a preferred received TPN; 

means for controlling the timing of each node, other 55 
than said one of said plurality of prioritized master 
nodes, from clock signals transmitted over the link 
that applies the preferred TPN to said controlled 
node; and 

means for controlling the timing of said one of said 60 
plurality of prioritized master nodes from clock 
signals transmitted over a connected link upon the 
TPN received from said connected link being less 
than M*N thereby allowing another one of said 
plurality of prioritized master nodes having higher 65 
priority to control the timing of said system upon 
said other one of said plurality of prioritized master 
nodes becoming operational 



5. The apparatus of claim 4 wherein said means of 
activating said one of said plurality of prioritized master 
nodes, comprises: 

means for detecting that all TPNs received by said 
one of said plurality of prioritized master nodes are 
equal to or greater than a predefined maximum 
value thereby indicating that none of said plurality 
of prioritized master nodes are controlling the tim- 
ing of said systems; and 

means for providing clock signals for controlling said 
multinode system in response to the detection that 
all TPNs are greater than said predefined maxi- 
mum value. 

6. The apparatus of claim 5 wherein said means of 
controlling the timing of each node, other than said 
plurality prioritized master nodes, comprises: 

means for detecting that all TPNs received by each 
node, other than said plurality of prioritized master 
nodes, are equal to or greater than said predefined 

maximum value; 

means for selecting an internal local oscillator to 
provide timing for each node in response to the 
detection of all TPNs being equal to or greater than 
said predefined maximum value; and 

means for transmitting a TPN equal to said prede- 
fined maximum value on each of said links con- 
nected to each node in response to the detection of 
all TPNs being equal to or greater than said prede- 
fined maximum value. 

7. The apparatus of claim 4 further comprises: means 
for initializing said other one of said plurality master 
nodes to control the timing of said system by generating 
and transmitting a TPN equal to M*N+i where M is 
the priority number of said other one of said plurality of 
prioritized master nodes. 

8. An apparatus for controlling the timing of each 
node in a multinode system wherein a clock source in 
one of a plurality of prioritized master nodes provides a 
reference clock signal for all other nodes in said system 
and the higher priority node of said plurality of priorit- 
ized master nodes has the lower priority number with 
the highest priority node having the lowest master pri- 
ority number, comprising: 

means for interconnecting each of said nodes to one 
or more other ones of said nodes via links; 

means for transmitting clock signals between each of 
said nodes directly interconnected by any one of 
said links; 

means for activating said plurality of prioritized mas- 
ter nodes to each provide reference clock signals 
for controlling the timing of said system; 

means for including a timing priority number (TPN) 
of M*N+i in the clock signals sent from each of 
said plurality of prioritized master nodes to all of 
said nodes directly connected to each of said plu- 
rality of prioritized master nodes by any one of said 
links where i is a predefined number M is the prior- 
ity number of each of said plurality of prioritized 
master nodes and N is equal to the number of links 
in the shortest path between the most distant nodes 
where distance is measured by counting the num- 
ber of links between nodes; 

means for operating all others of said nodes, other 
than said plurality of prioritized master nodes, that 
receives a TPN so that all others of said nodes alter 
the received TPN by adding the number i to the . 
received TPN and transmit the altered TPN with 
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clock signals to each node to which it is directly 
connected by any one of said links; 

means for operating each node that receives more 
than a single TPN, other than said plurality of 
prioritized master nodes, to select one of said re- 5 
ceived TPNs as a preferred received TPN; 

means for controlling the timing of each node, other 
than said plurality of prioritized master nodes, from 
clock signals transmitted over the link that applies 
the preferred TPN to said controlled node; and 10 

means for controlling the timing of each of said plu- 
rality of prioritized master nodes from clock signals 
transmitted over a connected link upon the TPN 
received from said connected link being less than 
M*N thereby allowing the one of said plurality of 15 
prioritized master nodes having the highest priority 
to gain control of the timing of said system. 

9. The apparatus of claim 8 comprises: 

means for detecting that all TPNs received by an- 
other one of said plurality of prioritized master 20 
nodes are equal to or greater than a predefined 
maximum value thereby indicating that none of 
said plurality of prioritized master nodes are con- 
trolling the timing of said system; and 

means for providing clock signals for controlling said 25 
system by said other one of said plurality of priorit- 
ized master nodes in response to the detection that 
all TPNs are greater than said predefined maxi- 
mum value, 

10. The apparatus of claim 9 wherein said means of 30 
controlling the timing of each node, other than said 
plurality of prioritized master nodes, comprises: 

means for detecting that all TPNs received by each 
node, other than said plurality of prioritized master 
nodes, are equal to or greater than said predefined 35 
maximum value; 

means for selecting an internal local oscillator to 
provide timing for each node in response to the 
detection of all TPNs being equal to or greater than 
said predefined maximum value; and 40 

means for transmitting a TPN equal to said prede- 
fined maximum value on each of said links con- 
nected to each node in response to the detection of 
all TPNs being equal to or greater than said prede- 
fined rrtflTifT^im value. 45 

11. An apparatus of controlling the timing of each 
node in a multinode system wherein a clock source in 
one of a plurality of prioritized master nodes provides a 
reference clock signal for all other nodes in said mul- 
tinode system and the higher priority node of said plu- 50 
rality of prioritized master nodes has the higher priority 
number with the highest priority node having the high- 
est priority number, comprising: 

means for interconnecting each of said nodes to one 
or more other ones of said nodes via links; 55 

means for transmitting clock signals between each of 
said nodes directly interconnected by any one of 
said links; 

means for activating said plurality of prioritized mas- 
, ter nodes to each provide reference clock signals 60 
for controlling the timing of said system; 

means for including a timing priority number (TPN) 
of M*N+a— i in the clock signals sent from each of 
said plurality of prioritized master nodes to all of 
said nodes directly connected to each of said plu- 65 
rality of prioritized master nodes by any one of said 

. links where a and i are predefined numbers and M 
is the priority number of each of said plurality of 
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prioritized master nodes and N is equal to the num- 
ber of links in the shortest path between the most 
distant nodes where distance is measured by count- 
ing the number of links between Tiodes; 

means for operating all others of said nodes, other 
than said plurality of prioritized master nodes, that 
receives a TPN so that all others of said nodes alter 
the received TPN by subtracting the number i from 
the received TPN and transmit the altered TPN 
with clock signals to each node to which it is di- 
rectly connected by any one of said links; 

means for operating each node that receives more 
than a single TPN, other than said plurality of 
prioritized master nodes, to select one of said re- 
ceived TPNs as a preferred received TPN; 

means for controlling the timing of each node, other 
than said plurality of prioritized master nodes, from 
clock signals transmitted over the link that applies 
the preferred TPN to said controlled node; and 

means for controlling the timing of each of said plu- 
rality of prioritized master nodes from clock signals 
transmitted over a connected link upon the TPN 
received from said connected link being greater 
than M*N-ha— 2*i thereby allowing the one of 
said plurality of prioritized master nodes having 
the highest priority to gain control of the timing of 
said multinode system. 

12. The apparatus of claim 11 comprises: 

means for detecting that all TPNs received by an- 
other one of said plurality of prioritized master 
nodes are equal to or less than a predefined mini- 
mum value thereby indicating that none of said 
plurality of prioritized master nodes are controlling 
the timing of said system; and 

means for providing clock signals for controlling said 
system by said other one of saidplurality of priorit- 
ized master nodes in response to the detection that 
all TPNs are equal to or less than said predefined 

minimum value. 

13. The apparatus of claim 12 wherein said means of 
controlling the timing of each node, other than said 
plurality of prioritized master nodes, comprises: 

means for detecting that all TPNs received by each 
node, other than said plurality of prioritized master 
nodes, are equal to or less than said predefined 

minimum value; 

means for selecting an internal local oscillator to 
provide timing for each node, other than said plu- 
rality of prioritized master nodes; and 

means for transmitting a TPN equal to said prede- 
fined minimum value on each of said links con- 
nected to each node, other than said plurality of 
prioritized master nodes. 

14. A method for of controlling the timing of each 
node in a multinode system wherein a clock source in 
one of plurality of prioritized master nodes provides a 
reference clock signal for all other nodes in said system 
and the higher priority node of said plurality of priorit- 
ized master nodes has the lower priority number with 
the highest priority node having the lowest priority 
number, said method comprising the steps of: 

(1) interconnecting each of said nodes to one or more 
other ones of said nodes via links; 

(2) transmitting clock signals between each of said 
nodes directly interconnected by any one of said 
links; 
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(3) activating one of said plurality of prioritized mas- 
ter nodes to provide reference clock signals for 
controlling the timing of said system; 

(4) including a timing priority number (TON) of 
M*N+i in the clock signals sent from said one of 5 
said plurality of prioritized master nodes to all of 
said nodes directly connected to said one of said 
plurality, of prioritized master nodes by any one of 
said links where i is a predefined number M is the 
priority number of said one of said plurality of 10 
prioritized master nodes and N is equal to the num- 
ber of links in the shortest path between the most 
distant nodes where distance is measured by count- 
ing the number of links between nodes; 

(5) operating all others of said nodes, other than said 15 
one of said plurality of prioritized master nodes, 
that receives a TPN so that all others of said nodes 
alter the received TPN by adding the number i to 
the received TPN and transmitting the altered 
TPN with clock signals to each node to which it is 20 
directly connected by any one of said links; 

(6) operating each node that receives more than a 
single TPN, other than said one of said plurality of 
prioritized master nodes, to select one of said re- 
ceived TPNs; 25 

(7) controlling the timing of each node, other than 
said one of said plurality of prioritized master 
nodes, from clock signals transmitted over the link 
that applies the selected TPN to said controlled 
node; and 30 

(8) controlling the timing of said one of said plurality 
of prioritized master nodes from clock signals 
transmitting over a connected link upon the TPN 
received from said connected link being less than 
M*N thereby allowing another one of said plural- 35 
ity of prioritized master nodes having higher prior- 
ity to control the timing of said system upon said 
other one of said plurality of prioritized master 
nodes becoming operational. 

15. The method of claim 14 wherein said step of acti- 40 
vating said one of said plurality of prioritized master 
nodes, comprises the steps of: 

(1) detecting that all TPNs received by said one of 
said plurality of prioritized master nodes are equal 

to or greater than a predefined maximum value 45 
thereby indicating that none of said plurality of 
prioritized master nodes are controlling the timing 
of said system; and 

(2) providing clock signals for controlling said system 

in response to the detection that all TPNs are 50 
greater than or equal to said predefined maximum 
value. 

16. The method of claim 15 wherein the step of con- 
trolling the timing of each node, other than said plural- 
ity of prioritized master nodes, comprises the steps of: 55 

(1) detecting that all TPNs received by each node, 
other than said plurality of prioritized master 
nodes, are equal to or greater than said predefined 
maximum value; 

(2) selecting an internal local oscillator to provide 60 
timing for each node in response to the detection 
that all received TPNs are equal to or greater than 
said predefined maximum value; and 

(3) transmitting a TPN equal to said predefined maxi- 
mum value on each of said links connected to each 65 
node in response to the detection that all received 
TPNs are equal to or greater than said predefined 
maximum value. 
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17. The method of claim 14 further comprises the step 
of: (1) initializing said other one of said plurality of 
prioritized master nodes to control the timing of said 
system by generating and transmitting a TPN equal to 
M*N+i where M is the priority number of said other 
one of said plurality of prioritized master nodes. 

18. A method for controlling the timing of each node 
in a multinode system wherein a clock source in one of 
a plurality of prioritized master nodes provides a refer- 
ence clock signal for all other nodes in said system and 
the higher priority node of said plurality of prioritized 
master nodes has the lower priority number with the 
highest priority node having the lowest priority num- 
ber, said method comprising the steps of: 

(1) interconnecting each of said nodes to one or more 
other ones of said nodes via Links; 

(2) transmitting clock signals between each of said 
nodes directly interconnected by any one of said 
links; 

(3) activating said plurality of prioritized master 
nodes to each provide reference clock signals for 
controlling the timing of said system; 

(4) including a timing priority number (TPN) of 
M*N+i in the clock signals sent from each of said 
plurality of prioritized master nodes to all of said 
nodes directly connected to each of said plurality 
of prioritized master nodes by any one of said links 
where i is predefined number and M is the priority 
number of each of said plurality of prioritized mas- 
ter nodes and N is equal to the number of links in 
the shortest path between the most distant nodes 
where distance is measured by counting the num- 
ber of links between nodes; 

(5) operating all others of said nodes, other than said 
plurality of prioritized master nodes, that receives a 
TPN so that all others of said nodes alter the re- 
ceived TPN by adding the number i to the received 
TPN and transmitting the altered TPN with clock 
signals to each node to which it is directly con- 
nected by any one of said links; 

(d) operating each node that receives more than a 
single TPN, other than said plurality of prioritized 
master nodes, to select one of said received TPNs 
as a preferred received TPN; 

(7) controlling the timing of each node, other than 
said plurality of prioritized master nodes, from 
clock signals transmitted over the link that applies 
the preferred TPN to said controlled node; and 

(8) controlling the tuning of each of said plurality of 
prioritized master nodes from clock signals trans- 
mitted over a connected link upon the TPN re- 
ceived from said connected link being less than 
M*N thereby allowing the one of said plurality of 
prioritized master nodes having the highest priority 
to gain control of the timing of said system. 

19. The method of claim 18 comprises the steps of: 

(1) detecting that all TPNs received by another one 
of said plurality of prioritized master nodes are 
equal to or greater than a predefined maximum 
value thereby indicating that none of said plurality 
of prioritized master nodes are controlling the tim- 
ing of said system; and 

(2) providing clock signals for controlling said system 
by said other one of said plurality of prioritized 
master nodes in response to the detection that all 
TPNs are greater than said predefined maximum 
value. 
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20. The method of claim 19 wherein the step of con- 
trolling the timing of each node, dther than said plural- 
ity of prioritized master nodes, comprises the steps of: 

(1) detecting that all TPNs received by each node, 
other than said plurality of prioritized master 
nodes, are equal to or greater than said predefined 
maximum value; 

(2) selecting an internal local oscillator to provide 
timing for each node in response to the detection of 
all TPNs being equal to or greater than said prede- 
fined maximum value; and 

(3) transmitting a TPN equal to said predefined maxi- 
mum value on each of said links connected to each 
node in response to the detection of all TPNs being 
equal to or greater than said predefined maximum 
value. 

21. A method of controlling the timing of each node 
in a multinode system wherein a clock source in one of 
a plurality of prioritized master nodes provides a refer- 
ence clock signal for all other nodes in said system and 20 
the higher priority node of said plurality of prioritized 
master nodes has the higher priority number with the 
highest priority node having the higher priority num- 
ber, said method comprising the steps of: 

(1) interconnecting each of said nodes to one or more 25 
other ones of said nodes via links; 

(2) transmitting clock signals between each of said 
nodes directly interconnected by any one of said 
links; 

(3) activating said plurality of prioritized master 30 
nodes to each provide reference clock signals for 
controlling the timing of said system; 

(4) including a timing priority number (TPN) of 
M*N-f a— i in the clock signals sent from each of 
said plurality of prioritized master nodes to all of 35 
said nodes directly connected to each of said plu- 
rality of prioritized master nodes by any one of said, 
links where a and i are predefined numbers and M 

is the priority number of each of said plurality of 
prioritized master nodes and N is equal to the num.- 40 
ber of links in the shortest path between the most 
distant nodes where distance is measured by count- 
ing the number of links between nodes; 

(5) operating all others of said nodes, other than said 
plurality of prioritized master nodes, that receives a 45 
TPN so that all others of said nodes alter the re- 
ceived TPN by subtracting the number i from the 



received TPN and transmit the altered TPN with 
clock signals to each node to which it is directly 
connected by any one of said links; 

(6) operating each node that receives more tha a. 
single TPN, other than said plurality of prioritized 
master nodes, to select one of said received TPNs 
as a preferred received TPN; 

(7) controlling the timing of each node, other than 
said plurality of prioritized master nodes, from 
clock signals transmitted over the link that applies 
the preferred TPN to said controlled node; and 

(8) controlling the timing of each of said plurality of 
prioritized master nodes from clock signals trans- 
mitted over a connected link upon the TPN re- 
ceived from said connected link being greater than 
M*N+a— 2*i thereby allowing the one of said 
plurality of prioritized master nodes having the 
highest priority to gain control of the timing of said 
system. 

22. The method of claim 21 comprises the steps of: 

(1) detecting that all TPNs received by another one 
of said plurality of prioritized master nodes are 
equal to or less than a predefined minimum value 
thereby indicating that none of said plurality of 
prioritized master nodes are controlling the timing 
of said system; and 

(2) providing clock signals for controlling said system 
by said other one of said plurality of prioritized 
master nodes in response to the detection that all 
TPNs are equal to or less than said predefined 
minimum value. 

23. The method of claim 22 wherein the step of con- 
trolling the timing of each node, other than said plural- 
ity of prioritized master nodes, comprises the steps of: 

(1) detecting that all TPNs received by each node, 
other than said plurality of prioritized master 
nodes, are equal to or less than said predefined 

minimum value; 

(2) selecting an internal local oscillator to provide 
timing for each node, other than said plurality of 
prioritized master nodes; and 

(3) transmitting a TPN equal to said predefined mini- 
mum value on each of said links connected to each 
node, other than said plurality of prioritized master 
nodes. 
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